Portable terminal apparatus for capturing more than one image of identical object, and captured image processing system for obtaining high resolution image data based on the captured images and outputting high resolution image

ABSTRACT

A portable terminal apparatus consecutively more than once captures images of an identical image capture object. The portable terminal apparatus determines whether or not a plurality of pieces of captured image data captured include a combination of a given number (an integer not less than 2) of pieces of captured image data which are applicable to be subjected to the high resolution correction by the correction processing section and are offset by a given amount, and transmits, to an image output apparatus, the given number of pieces of captured image data which are determined to be the combination. The image output apparatus receives the given number of pieces of captured image data from the portable terminal apparatus, and carries out, in accordance with the given number of pieces of captured image data, the high resolution correction for preparing high resolution image data which has a higher resolution than the given number of pieces of captured image data received. Thereafter, the image output apparatus carries out an output process for outputting the high resolution image data prepared. This provides a captured image processing system which allows an image output apparatus to output an image captured by a portable terminal apparatus while the image is improving in resolution such as text readability.

This Nonprovisional application claims priority under 35 U.S.C. §119(a)on Patent Application No. 2009-091530 filed in Japan on Apr. 3, 2009,the entire contents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a captured image processing systemwhich causes an image output apparatus to output an image captured by aportable terminal apparatus.

2. Description of the Related Art

A mobile phone and wire and wireless technologies have been developedand widespread in tandem with advancement in Internet technology. Thetechnologies make it possible that an image is transmitted, in a placewhere a user is staying, from a portable terminal apparatus such as themobile phone to an image output apparatus connected to the Internet soas to be put into print.

For example, according to Patent Literature 1, digital image data,collected by a digital camera or a mobile information device such as acamera-equipped PDA or a mobile personal computer, is transmitted to aserver via a network. The server carries out image processing such as abrightness adjustment, a color adjustment, and/or a size adjustment withrespect to the digital image data. Then, the server embeds the digitalimage data in an image area, in which a plurality of document combiningtemplates are stored, so as to prepare a document.

According to Patent Literature 2, in a camera-equipped mobile phone, atarget image to be printed is selected, a variety of printingrequirements are set, printing is instructed to start, andidentification information (a) such as a device name of the mobile phoneis transmitted to a printer so as to be registered with the printer.After receiving the identification information (a), the printertransmits identification information (b) such as a device name of theprinter to the mobile phone so that the identification information (b)is registered with the mobile phone.

Next, a parameter of a communication speed which is usable for datacommunication is sent to the printer from the mobile phone. The printercompares the parameter with a usable communication speed for theprinter. The printer sends, to the mobile phone, a parameter of thecommunication speed which parameter is determined to be the mostefficient in transmission. The mobile phone alters its communicationspeed to a communication speed received from the printer, and theprinter also alters its communication speed to the communication speedtransmitted to the mobile phone. Then, the mobile phone transfers imagedata to the printer, and the printer receives the image data so as toprepare print data and carry out a printing process.

Citation List

Patent Literature 1

Japanese Patent Application Publication, Tokukai, No. 2002-41502 A(Publication Date: Feb. 8, 2002)

Patent Literature 2

Japanese Patent Application Publication, Tokukai, No. 2007-27839 A(Publication Date: Feb. 1, 2007)

SUMMARY OF INVENTION Technical Problem

However, an RGB filter or a complementary color (CMY) filter, which isused in a mobile phone or a digital camera, is arranged such thatfilters are provided for respective colors in a two-dimensional manner.Therefore, an image captured by a portable terminal apparatus has aresolution which is approximately one third to two thirds lower thanthat of image data scanned by a scanner or the like. This causes, inparticular, a problem that the image has poor text readability.

Specifically, an RGB line sensor is used in the scanner or the like.This allows a piece of color data to be scanned for a correspondingpixel. In contrast, according to the mobile phone or the digital camera,(i) filters RGBRGB . . . are provided for respective pixels or (ii)filters RGRGRG . . . /GBGBGB . . . are provided for each line of pixels.This causes a color component for which no filter is provided to existin each of the pixels, thereby causing a low resolution.

The present invention has been made in view of the problems, and itsobject is to provide a portable terminal apparatus, an image outputapparatus, a capture image processing system, a method for controllingthe portable terminal apparatus, an image output method, and a recordingmedium, the image output apparatus being capable of outputting an imagecaptured by the portable terminal apparatus while the image is improvingin resolution such as text readability.

Solution to Problem

In order to attain the object, a portable terminal apparatus of thepresent invention for transmitting image data to an image outputapparatus, the image output apparatus including: a correction processingsection for carrying out a high resolution correction in accordance witha plurality of pieces of the image data so as to obtain high resolutionimage data which has a higher resolution than the plurality of pieces ofthe image data, the image output apparatus outputting the highresolution image data corrected by the correction processing section,said portable terminal apparatus, includes: an image capture section forconsecutively more than once capturing images of an identical object; acaptured image determination section for determining whether or not aplurality of pieces of captured image data captured by the image capturesection meet a given requirement; and a transmission section fortransmitting, to the image output apparatus, captured image data whichis determined, by the captured image determination section, to meet thegiven requirement, the given requirement including a requirement A, therequirement A being such that the plurality of pieces of captured imagedata consecutively captured by the image capture section include acombination of a given number (an integer not less than 2) of pieces ofcaptured image data which are applicable to be subjected to the highresolution correction by the correction processing section and areoffset by a given amount, the transmission section transmitting thegiven number of pieces of captured image data which are determined to bethe combination by the captured image determination section.

An image output apparatus of the present invention includes: a receivingsection for receiving, from a portable terminal apparatus as mentionedabove, the given number of pieces of captured image data obtained fromthe image capture which is consecutively more than once carried out bythe image capture section; a correction processing section for carryingout a high resolution correction for preparing, in accordance with thegiven number of pieces of captured image data received by the receivingsection, high resolution image data which has a higher resolution thanthe given number of pieces of captured image data; and an output sectionfor carrying out an output process in which the high resolution imagedata prepared by the correction processing section or an image indicatedby the high resolution image data is outputted.

Advantageous Effects of Invention

According to the present invention, it is possible to cause an imageoutput apparatus to output an image captured by a portable terminalapparatus while the image is improving in resolution such as textreadability.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart illustrating a captured image processing methodcarried out in a captured image processing system of the presentinvention. (a) of FIG. 1 illustrates a flow in a portable terminalapparatus, and (b) of FIG. 1 illustrates a flow in an image outputapparatus.

FIG. 2 is a drawing illustrating an arrangement of the captured imageprocessing system of the present invention.

FIG. 3 is a block diagram illustrating an arrangement of the portableterminal apparatus of the present invention.

FIG. 4 illustrates an example of detection of a skew of an image whichdetection is carried out in the captured image processing method of thepresent invention.

FIG. 5 shows angles of a skew θ and their respective tangents whichangles and tangents are obtained in the example of detection of the skewwhich example is illustrated in FIG. 4.

FIG. 6 illustrates an example of detection of a geometric distortion ofan image.

FIG. 7 illustrates an example of an edge detection process carried outwith respect to an object in an image.

FIG. 8 illustrates an example of detection of an edge of an image in araster direction.

FIG. 9 shows an example of a first order differential filter used in anexample of detection of a degree of offset between images.

FIG. 10 is a block diagram illustrating an arrangement of the imageoutput apparatus of the present invention.

FIG. 11 illustrates an example of a look-up table used during detectionof a color balance of an image.

FIG. 12 illustrates an example of a correction for a lens distortion ofan image.

FIG. 13 illustrates an example of a correction for the geometricdistortion of an image.

FIG. 14 illustrates an example of determination of a reconstructed pixelvalue of an image.

DETAILED DESCRIPTION OF THE INVENTION

An embodiment of the present invention is described below in detail.

(1) Overall Arrangement of Captured Image Processing System

FIG. 2 is a drawing illustrating an overall arrangement of a capturedimage processing system of the present invention. The captured imageprocessing system includes (i) a portable terminal apparatus 100including image capture means such as a camera-equipped mobile phone ora digital camera and (ii) an image output apparatus 200 such as amultifunction printer or a printer (an image forming apparatus) (seeFIG. 2).

The portable terminal apparatus 100 is carried with a user. The user cancause the portable terminal apparatus 100 to carry out image capturewith respect to an object in various scenes. However, an image capturedby the portable terminal apparatus 100 has a low resolution as describedearlier. Therefore, even in a case where an image having such a lowresolution is subjected to an image output process (e.g., printing) bythe image output apparatus 200, it may be impossible to recognizedetails of the image. For example, in a case where image capture iscarried out with respect to (a) paper or a poster on which a text image(e.g., a text) is printed or (b) a display screen on which the textimage or the like is displayed, the text which has been subjected to theimage output process may be illegible. According to the presentembodiment, it is possible to obtain an image which has a higherresolution than an image captured by the portable terminal apparatus100, even in such a case.

According to the present embodiment, the portable terminal apparatus 100has a function of a text image capture mode in which it is possible toobtain, from the image output apparatus 200, an image which has a higherresolution than an actually captured image.

Note, in the present embodiment, that the user is assumed to select thetext image capture mode, in a case where (i) the user carries out imagecapture with respect to an image capture object (an object whose imageis to be captured) which has a rectangular shape such as (a) paper or aposter on which a text image is printed or (b) a display screen on whichthe text image is displayed (e.g., a display screen and a screenprojected by a projector) and (ii) desires to obtain the image which hasa higher resolution than the actually captured image.

Note also that it is not always possible for the user to carry out imagecapture from the front with respect to the image capture object whichhas a rectangular shape such as (a) paper or a poster on which a textimage is printed or (b) a display screen on which the text image isdisplayed. Namely the user may obliquely carry out image capture withrespect to the image capture object, in a state where (i) a normaldirection of a plane of the image capture object on which plane the textimage is formed and (ii) a direction in which image capture meanscarries out the image capture do not coincide with each other. In thiscase, the image capture object undergoes a distortion (hereinafterreferred to as a geometric distortion) in the captured image. Thepresent embodiment is arranged to cause the image output apparatus 200to output an image, in which such a geometric distortion has beencorrected, in a case where the text image capture mode is selected.

The portable terminal apparatus 100, which can communicate with theimage output apparatus 200, transmits data of the captured image(hereinafter referred to as captured image data) to the image outputapparatus 200.

The image output apparatus 200 carries out a high resolution correctionfor resolution enhancement with respect to the captured image inaccordance with a plurality of pieces of the captured image data on anidentical image capture object which are received from the portableterminal apparatus 100. The image output apparatus 200 carries out anoutput process with respect to the captured image whose resolution hasbeen enhanced. It is possible to employ a method disclosed in theJournal of the Institute of Image Information and Television EngineersVol. 62, No. 3, pp. 337 through 342 (published in 2008) as a method forcarrying out the high resolution correction by use of the plurality ofpieces of the captured image data.

Examples of the output process carried out by the image output apparatus200 include (i) a printing process carried out with respect to an imagein accordance with the captured image data whose resolution has beenenhanced, (ii) a filing process for causing the captured image datawhose resolution has been enhanced to be stored in a storage device suchas a server or a USB memory, and (iii) an e-mail transmission processfor transmitting an e-mail to which the captured image data whoseresolution has been enhanced is attached.

The portable terminal apparatus 100 can be communicated with the imageoutput apparatus 200 as below. The captured image data is transferredfrom the portable terminal apparatus 100 to the image output apparatus200 via a wireless communication system which is in conformity with anyone of the infrared communication standards such as IrSimple (see a signA illustrated in FIG. 2). Alternatively, the captured image data istransmitted from the portable terminal apparatus 100 temporarily to anintermediary apparatus 300 via a non-contact wireless communicationsystem such as Felica (registered trademark) (see a sign B illustratedin FIG. 2) and then transferred from the intermediary apparatus 300 tothe image output apparatus 200 via a wireless communication system suchas Bluetooth (registered trademark) (see a sign C illustrated in FIG.2). Note that not only the communication systems but also a systememploying a publicly-known method is applicable to the communicationbetween the portable terminal apparatus 100 and the image outputapparatus 200.

(2) Arrangement of the Portable Terminal Apparatus

First, the portable terminal apparatus 100 of the present embodiment isdescribed below with reference to FIGS. 3 through 9.

FIG. 3 is a block diagram illustrating an arrangement of the portableterminal apparatus 100. The portable terminal apparatus 100 includes animage capture section 101, a captured image determination section 102,an image processing section 103, a communication section (transmissionsection) 104, a display section (notification section) 105, an inputsection 106, a recording medium accessing section 107, a storage section108, and a control section (transmission section, notification section)109.

The image capture section 101 carries out image capture with respect toan image capture object by use of a CCD/CMOS sensor so that a capturedimage has a predetermined resolution. In a case where the text imagecapture mode is selected by the user, a single shutter click causes theimage capture section 101 to consecutively carry out, more than once(e.g., 2 to 15 times), image capture with respect to the image captureobject. Images consecutively captured are generally substantiallyidentical, but will be offset by a minutely small amount due to a camerashake or the like.

While the text image capture mode is being selected, the captured imagedetermination section 102 determines whether or not a plurality ofcaptured images captured by the image capture section 101 meet processexecution requirements including a requirement for causing the imageoutput apparatus 200 to carry out the high resolution correction. Thecaptured image determination section 102 supplies a determined result tothe control section 109. Processes carried out by the captured imagedetermination section 102 are described later in detail.

The image processing section 103 carries out at least an A/D conversionprocess with respect to the data of the image captured by the imagecapture section 101 (captured image data).

The communication section 104 has functions of serial/parallel transferand wireless data communication which are in conformity with USB 1.1 orUSB 2.0 Standard. The communication section 104 transmits to the imageoutput apparatus 200 a plurality of pieces of the data of the imageswhich are captured by the image capture section 101 and are thensubjected to the A/D conversion process by the image processing section103. Note, however, that the communication section 104 transmits onlythe captured image data that is determined, by the captured imagedetermination section 102, to meet the process execution requirements.

The display section 105 is realized by a liquid crystal display device,for example. The input section 106, which has a plurality of buttons,serves as a section from which the user enters data.

The storage medium accessing section 107 reads out a program forcarrying out the processes in the portable terminal apparatus 100 from arecording medium in which the program is recorded.

The storage section 108 serves as a section in which (i) the program forcarrying out the processes in the portable terminal apparatus 100, (ii)information on a model of the portable terminal apparatus 100, (iii)user information, and (iv) data required for carrying out the processesare stored. Note that the user information refers to information, foridentifying the user of the portable terminal apparatus 100, such as auser ID and a password.

The control section 109 carries out control with respect to the sectionsof the portable terminal apparatus 100. Specifically, in a case where aninstruction to select the text image capture mode is entered to theinput section 106, the control section 109 causes the display section105 to display a window which urges the user to enter, from the inputsection 106, a magnification of resolution conversion. Subsequently, thecontrol section 109 determines, in accordance with the magnification(e.g., ×2 or ×4) entered from the input section 6, (i) the number ofconsecutive times of image capture carried out by the image capturesection 101 and (ii) a part of the process execution requirements whichis used in the captured image determination section 102. Note that thecontrol section 109 determines the above (i) and (ii) in accordance withinformation, preliminarily stored in the storage section 108, in whichthe magnification, the number of times of image capture, and the part ofthe process execution requirements are associated with each other.

Further, in the case where the instruction to select the text imagecapture mode is entered from the input section 106, the control section109 causes the display section 105 to display a window which urges theuser to enter, from the input section 106, (i) an instruction to selecta kind of the output process (such as the printing process, the filingprocess, the e-mail transmission process, or the like) and (ii) asetting requirement for carrying out a selected output process (aprinting requirement such as the number of sheets to be printed, anaddress of a server at which data is to be filed, an address of adestination at which an e-mail is transmitted, or the like).Subsequently, the control section 109 receives output processinformation indicative of the kind of the output process and the settingrequirement for carrying out the output process.

The control section 109 attaches, to the captured image data which isdetermined, by the captured image determination section 102, to meet theprocess execution requirements, (i) a file name, (ii) the information onthe model of the portable terminal apparatus 100 and the userinformation which are stored in the storage section 108, and (iii) theoutput process information. Thereafter, the control section 109 causesthe communication section 104 to carry out a process of transmittingthis captured image data to the image output apparatus 200.

(3) Processes carried out by the Captured Image Determination Section

The following description discusses how the captured image determinationsection 102 of the portable terminal apparatus 100 carries out thedetermination processes.

(3-1) Determination of Skew

As described earlier, the user selects the text image capture mode in acase where the user carries out image capture with respect to the imagecapture object, which has a rectangular shape, such as paper, a poster,or a display screen and desires to obtain a high resolution image.Therefore, the captured image determination section 102 assumes that theimage capture object has a rectangular shape, and detects, in thecaptured image data, a skew of the image capture object by detecting anedge of the image capture object. Note that a conventionally knownmethod can be employed as a method for detecting, in the captured imagedata, a pixel located on the edge of the image capture object which hasa rectangular shape. In order to prevent a background edge from beingerroneously determined to be the edge of the image capture object, it isalternatively possible to employ a method in which it is determined thatan edge of the image capture object is detected only in a case where anedge having a length of not less than a given length is detected. Inthis case, the given length can be set, for example, to a length whichis approximately 80% of a length of an end side of an image in thecaptured image data. Alternatively, it is also possible to cause theuser to select the edge of the image capture object from the edges thusdetected. It is possible to employ, as such an edge detection method, atechnique disclosed in Japanese Patent Application Publication, Tokukai,No. 2006-237757 A.

The captured image determination section 102 selects two points locatedon the detected edge of the image capture object. For example, thecaptured image determination section 102 selects two points 11 and 12which are away from a center of the captured image data by w/2 in atransverse direction to the right and left, respectively (see FIG. 4).Next, it is possible to determine a skew of the image capture object inthe captured image by determining shortest distances d₁ and d₂ betweenan end side of the captured image data and the respective selected twopoints 11 and 12. In the case of FIG. 4, when an angle of the skew isindicated as θ, tan θ=(d₂−d₁)/w. Then, the captured image determinationsection 102 calculates a value of (d₂−d₁)/w and reads out acorresponding angle θ, for example, from a table (refer to FIG. 5) whichhas been prepared in advance.

Subsequently, the captured image determination section 102 determineswhether or not the detected angle θ falls within a given range (e.g.,−30° to +30° and supplies a determined result to the control section109. Note here that it is one of the process execution requirements thatthe angle θ falls within the given range.

(3-2) Determination of Geometric Distortion

As described earlier, the geometric distortion means that in a casewhere image capture is obliquely carried out with respect to the imagecapture object from a direction different from the normal direction ofthe plane of the image capture object on which plane the text image isformed, the image capture object has, in the captured image, a distortedshape instead of the rectangular shape. For example, in a case whereimage capture is carried out with respect to the image capture objectobliquely, i.e., from a left below direction with respect to a normaldirection of the paper, the image capture object has a distortedquadrangular shape (see FIG. 6).

As described later, according to the present embodiment, the imageoutput apparatus 200 has a function of correcting such a geometricdistortion. Note, however, that in a case where the geometric distortionoccurs to a large degree, readability will not be so enhanced even ifthe geometric distortion is corrected. In view of this, the capturedimage determination section 102 of the present embodiment detectsfeatures indicative of a degree of the geometric distortion so as todetermine whether or not the features fall within a given range.

Edges of respective sides of the image capture object do not necessarilyexist in the vicinity of a center of the angle of view. In view of this,according to the present embodiment, edges are extracted, at givenregular intervals, from all sides, line segments identified by therespective edges are found, and intersections of these line segments arefound, thereby defining a region where the image capture object islocated.

First, the captured image determination section 102 carries out a rasterscanning with respect to the captured image data. Note here that (i) aforward direction and (ii) a direction which is perpendicular to theforward direction are an X direction and a Y direction, respectively(see FIG. 6). Note also that an upper left corner is an origin in thecaptured image.

In a case where no edge is detected as a result of the scanning carriedout with respect to one (1) line, the captured image determinationsection 102 carries out the scanning with respect to a subsequent linewhich is away from the one line by a predetermined distance in the Ydirection. Note that an interval between the lines is not limited to aspecific one, provided that it is a fixed one. Further, the line is notnecessarily constituted by a single pixel.

Next, in the raster scanning, the captured image determination section102 regards, as L₁ (a first line), a line on which an edge is firstlydetected. The captured image determination section 102 classifies, intoa first group, coordinates of a point determined to be the first edge inthe forward direction, and then classifies, into a second group,coordinates of a point determined to be the second edge on the firstline (see FIG. 7). The scanning is consecutively carried out withrespect to a subsequent line so that an edge is detected. Then, withrespect to each line L_(i), a difference in X-coordinate value between(a) a point firstly determined to be an edge of the image capture objectin the forward direction and (b) a point secondly determined to be anedge of the image capture object in the forward direction (a distanced_(i) between X-coordinates of the two points) is found, and then anedge determination is carried out as below.

It is assumed that the X-coordinate of the first edge on the line L_(i)is X_(i1) (the X-coordinate belonging to the first group) and theX-coordinate of the second edge on the line L_(i) is X_(i2) (theX-coordinate belonging to the second group). The features detectionmethod is carried out as below.

(a) Coordinates X₁₁ and X₁₂ on the first line (L₁) are invariable.

(b) As for an ith line (i is an integer of not less than 2), anintercoordinate distance d_(i1)(=X_(i1)−X_((i−1)1)) andd_(i2)(=X_(i2)−X_((i−1)2)) are found. Note that the followingdescription discusses d_(i1), and so omits a suffix 1. Same applies tod_(i2).

(c) As for an ith line (i is an integer of not less than 3),dd_(i)=abs{(d_(i))−d_(i−1)} is found. In a case where dd_(i)≦th₁ (≈asmall value close to 0 (zero)), a coordinate X_(i) is classified into anidentical group (the first group or the second group). Otherwise (in acase where dd_(i)>th₁), the coordinate X₁ is classified into a differentgroup (a third group or a fourth group).

(d) Only in a case where i=4, a process for deciding a group of X₂ iscarried out as an initial process. The process is carried out as below.

i) dd₃≦th₁ and dd₄≦th₁→X₂: identical group

ii) dd₃>th₁ and dd₄≦th₁→X₂: different group

iii) dd₃≦th₁ and dd₄>th₁→X₂: identical group

iv) dd₃>th₁ and dd₄>th₁→X₂: identical group

Once a transition of X₂ to the different group (the third group or thefourth group) occurs, it is unnecessary to check increase and decreasein dd_(i).

Such a process is carried out with respect to an entire image so thatedge points are extracted for each of the groups. Then, coordinates ofthe edge points which belong to each of the groups are subjected tolinearization by use of a method such as a method of least squares orthe like. This allows a straight line, which is approximate to the edgepoints which belong to each of the groups, to be found. The linescorrespond to the sides of the image capture object.

FIG. 8 is a drawing illustrating a case where edge points are extractedby the raster scanning in accordance with a process as mentioned aboveand classified into the four groups. Note, in FIG. 8, that a circleindicates an edge which belongs to the first group, a quadrangleindicates an edge which belongs to the second group, a triangleindicates an edge which belongs to the third group, and a star indicatesan edge which belongs to the fourth group. Note also in FIG. 8 thatstraight lines, which have been subjected to the linearization by use ofthe method of least squares so as to be approximate to the edge pointsfor each of the groups, are illustrated by respective dotted lines.

Then, intersections (intersections 1 through 4 illustrated in FIG. 8) ofthe straight lines for the respective four groups are found. This makesit possible to define a region surrounded by the four straight lines asa region where the image capture object is located.

Further, a classifying process as mentioned above can be carried outwith respect to an image which has been subjected to a 90-degreerotation. This also allows an extraction of edges of a document which isideally provided so as to be parallel to a horizontal direction and avertical direction of the image. Namely, the raster scanning allows adetection of an edge in the vertical direction in the image which hasnot been rotated. In contrast, the raster scanning allows a detection ofan edge which was in the horizontal direction before the image wasrotated (which is in the vertical direction after the image is rotated)in the image which has been rotated. This also allows an extraction ofedges which are parallel to the vertical direction and the horizontaldirection. As long as a sufficient amount of information is obtained(for example, not less than three intersections are obtained in each ofthe groups) before the rotation of the image, only this information canbe used. In contrast, in a case where the number of intersectionsobtained is less than one in any one of the groups, it is obviouslyimpossible to formulate a straight line. In such a case, intersectionsobtained after the rotation of the image can be used

Alternatively, it is also possible to formulate a straight line by (i)carrying out again a coordinate conversion with respect only to foundcoordinates of an intersection, (ii) finding a corresponding group fromregions in which the respective groups are distributed, and (iii)integrating information on the intersections. Namely, the straight linecan be formulated by integrating coordinates of intersections, whichbelong to an identical group, out of (i) coordinates of intersectionswhich coordinates are found by the image which has not been rotated and(ii) coordinates of intersections which coordinates are obtained bycarrying out a coordinate conversion with respect to intersections foundby the image which has been rotated.

Note that it is possible to extract an edge point in accordance with thefollowing method. Pixel values, obtained in a small window which has awidth of at least one pixel, are compared as they are (a sum or anaverages of the pixel values are compared in a case where the width isnot less than two pixels). In a case where pixel values of adjacentwindows have a difference of not less than a given value, an edge pointcan be determined. In order to prevent a background edge or an edge of atext included in the image capture object from being erroneouslydetermined to be the edge of the image capture object, it isalternatively possible to employ a method in which it is determined thatan edge of the image capture object is detected only in a case where anedge having a length of not less than a given length is detected. Inthis case, the given length can be set, for example, to a length whichis approximately 80% of a length of an end side of an image in thecaptured image data. Alternatively, it is also possible to cause theuser to select the edge of the image capture object from the edges thusdetected. It is possible to employ, as such an edge detection method, atechnique disclosed in Japanese Patent Application Publication, Tokukai,No. 2006-237757 A. Alternatively, it is also possible to prevent such anerroneous detection by carrying out an evaluation of each of thecoordinate groups or a process for detecting a line segment (e.g., aHough transformation). Further, it is possible to prevent an edge of atext or a fine texture from being erroneously detected by carrying out aprocess employing a reduced image as preprocessing.

After finding the four straight lines and their intersections, thecaptured image determination section 102 finds each ratio betweenlengths of opposite sides of the quadrangle defined by the four straightlines. The each ratio between the lengths can be easily found by use ofthe coordinates of the intersections. Note that the quadrangle has twopairs of the opposite sides and thus the captured image determinationsection 102 finds a ratio between lengths for each of the two pairs.

Note here that the ratio between the lengths of the opposite sides isequal to 1 (one to one) in a case where image capture is carried out,from the front, with respect to the image capture object which has arectangular shape, the image capture object included in the capturedimage also has a rectangular shape. In contrast, in a case where imagecapture is obliquely carried out with respect to the image captureobject which has a rectangular shape, the ratio becomes a valuedifferent from 1. This is because the image capture object included inthe captured image has a distorted quadrangular shape. As a direction inwhich image capture is carried out is at a greater angle to the normaldirection of the plane of the image capture object on which plane thetext image is formed, a difference between a value of the ratio and 1increases. It follows that the ratio between the lengths of the oppositesides is one of the features indicative of a degree of the geometricdistortion.

Then, the captured image determination section 102 determines whether ornot each of the two ratios that has been found falls within a givenrange (e.g., 0.5 to 2) and supplies a determined result to the controlsection 109. Note here that the given range is set in advance so that ageometric distortion correction can be made by the image outputapparatus 200, and is stored in the storage section 108. Note also thatit is one of the process execution requirements that each of the tworatios falls within the given range (e.g., 0.5 to 2).

Note that the captured image determination section 102 can use, asalternative features indicative of the degree of the geometricdistortion, an angle formed by two straight lines which are defined bytwo and the remaining two, respectively, of the four intersections whichhave been detected as above.

(3-3) Determination of Offset Amount of a Plurality of Images

As described earlier, the image output apparatus 200 carries out thehigh resolution correction in accordance with the plurality of pieces ofcaptured image data of the identical image capture object. In order tocarry out the high resolution correction, it is necessary that a givennumber of pieces of image data which varies depending on themagnification of resolution conversion be offset by a given amount. Inview of this, the captured image determination section 102 of thepresent embodiment determines whether or not the plurality of pieces ofcaptured image data (data of the images captured by the image capturesection 101) include a given number of pieces of the captured image datawhich are required to carry out the high resolution correction and whichare offset by a given amount.

Note that an offset, required for the high resolution correction whichallows enhancement of text readability, intends an offset of less thanone pixel (a decimal point) of target image data. Namely, an offset,which is below the decimal point (less than one pixel) such as thatfalls in a range of 0.3 to 0.7, is important. An offset corresponding toan integer part is not considered during the high resolution correction.For example, in the case of an offset corresponding to 1.3 pixel, 2.3pixels, or the like each including an offset of less than one pixel, itis possible to carry out the high resolution correction in accordancewith a plurality of images. In contrast, in the case of an offset of onepixel, two pixels, or the like each including no offset of less than onepixel, it is impossible to carry out the high resolution correction.

For example, in the case of a conversion magnification of ×2, the numberof pieces of image data which is required for the high resolutioncorrection is two (2). An offset amount of the decimal point of the twopieces of image data preferably falls in a range of 0.3 to 0.7, each ofwhich is a result obtained when the offset is represented by a pixel.Therefore, information in which (i) a magnification of the resolutionconversion “×2”, (ii) the number of times of image capture “2”, and(iii) a process execution requirement “required number of pieces ofimage data: 2, offset amount: 0.3 to 0.7” are associated with each otheris beforehand stored in the storage section 108. In accordance with theinformation, the control section 109 controls (i) the image capturesection 101 to carry out image capture two consecutive times and (ii)the captured image determination section 102 to carry out adetermination in accordance with the process execution requirement“required number of pieces of image data: 2, offset amount: 0.3 to 0.7”.

In the case of a conversion magnification of ×4, the number of pieces ofimage data which is required for the high resolution correction is 4. Ina case where one of the four pieces of data is assumed to be referenceimage data, amounts of offset of the decimal point of the other threepieces of image data with respect to the reference image data preferablyfall in ranges of 0.2 to 0.3, 0.4 to 0.6, and 0.7 to 0.8, respectively,each of which is a result obtained when the offset is represented by apixel. Therefore, information in which (i) a magnification of theresolution conversion “×4”, (ii) the number of times of image capture“4”, and (iii) a process execution requirement “required number ofpieces of image data: 4, offset amount: 0.2 to 0.3, 0.4 to 0.6, and 0.7to 0.8” are associated with each other is beforehand stored in thestorage section 108.

Note that the following description discusses, for simplicity, a case inwhich the magnification of the resolution conversion “×2” is selected.

First, the captured image determination section 102 selects any one ofthe captured images. As for the selected captured image (hereinafterreferred to as a first captured image), the captured image determinationsection 102 selects an offset detecting partial region from the regionwhich is defined during the determination of the geometric distortionand in which the image capture object is located. Note here that theoffset detecting partial region is used so that offset amounts of theremaining captured images (hereinafter referred to as a second capturedimage) with respect to the first captured image are found. Therefore, itis preferable to select the offset detecting partial region in whichthere occurs a great change in pixel value (there exists a clearpattern). As such, the captured image determination section 102 extractsthe offset detecting partial region in accordance with the followingmethod.

The captured image determination section 102 specifies a pixel, servingas a target pixel, existing in a centroid of the region where the imagecapture object is located. Subsequently, the captured imagedetermination section 102 selects a region where n×n pixels includingthe target pixel are provided. The captured image determination section102 judges whether or not the selected region satisfies the followingselection requirement. In a case where the selected region satisfies theselection requirement, the region becomes the offset detecting partialregion. In contrast, in a case where the selected region satisfies noselection requirement, the captured image determination section 102selects another region in accordance with a given offset and carries outan identical determination with respect to the another region. This ishow the offset detecting partial region is extracted.

Note here that examples of the selection requirement include thefollowing two requirements.

According to the first example of the selection requirement, a valuewhich is based on a variance obtained in the region is used. A variance(x) obtained in the offset detecting partial region is expressed as thefollowing expression (1), where P (i) is a pixel value of a region, inthe vicinity of the target pixel, in which region n×n pixels areprovided. The selection requirement is met when the variance (x) is notless than a given threshold. For simplicity, only a numerator of theexpression (1) can be considered.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{{{Varience}(x)} = \frac{{n \times {\sum\limits_{i = 0}^{n - 1}\left\lbrack {P(i)} \right\rbrack^{2}}} - \left\lbrack {\sum\limits_{i = 0}^{n - 1}{P(i)}} \right\rbrack^{2}}{n \times n}} & {{Expression}\mspace{14mu}(1)}\end{matrix}$

According to the second example of the selection requirement,binarization is carried out, by an edge extraction filter such as afirst order differential filter, with respect to the region, in thevicinity of the target pixel, in which region n×n pixels are provided,and a sum total of binarized values is used. FIG. 9 shows an example ofthe first order differential filter. Similar to the first example of theselection requirement, the second selection requirement is met when thesum total is not less than a given threshold (e.g., not less than 5% ofthe number of pixels in the offset detecting partial region).

Next, in contrast to an offset detecting partial image A (n×n) of thefirst captured image, an offset detecting partial image B (m×m) (m>n) iscut out from the second captured image, the offset detecting partialimage B having a center substantially identical to that of the offsetdetecting partial image A. The offset detecting partial image B is cutout so that coordinates of a central pixel of the offset detectingpartial image A in the first captured image coincide with coordinates ofa central pixel of the offset detecting partial image B in the secondcaptured image.

Then, a region of the clipped offset detecting partial image B whichregion best matches the offset detecting partial image A is found withsub-pixel-level accuracy. This can be realized by employing a normalizedcorrelation pattern matching in which the offset detecting partial imageA serves as a template.

As an example of the normalized correlation pattern matching, acorrelation is found by use of a well-known normalized correlationequation. A correlation equation of two patterns of Input (I) and Target(T) which include N pixels can be generally expressed as the followingexpression (2). Note here that α, β, and γ can be expressed as below.

[Math. 2]S={α/√{square root over (β×γ)}}  Expression (2)

[Math. 3]α=NΣ(I×T)−(ΣI)×(ΣT)β=NΣ(I×I)−(ΣI)×(ΣI)γ=NΣ(T×T)−(ΣT)×(ΣT)

A correlation value map of 3×3 is obtained, in a case where, for exampleunder the requirement of n=5 and m=7, the above correlation equation iscalculated for each region (n×n) of the offset detecting partial image B(m×m), which each region has an identical size to the offset detectingpartial image A. A fitting quadric surface is found by use of thecorrelation value map. The quadric surface is found based on an equationS(x,y)=a×x×x+b×x×y+c×y×y+d×x+e×y+f. Specifically, six points each ofwhich has a higher correlation value are selected from nine points, andsimultaneous equations are solved so that each coefficient is found. Itis determined that the process execution requirement “required number ofpieces of image data: 2, offset amount: 0.3 to 0.7” is met, in a casewhere values below the decimal point of coordinate values (both x and y)of an extreme value (=a maximum value) of the function S(x, y) fallwithin the given range (here, 0.3 to 0.7).

Note that an extreme value can be found by (i) carrying out partialdifferentiation with respect to the quadratic equation S(x, y), and then(ii) finding coordinates of a point where a corresponding partialdifferential coefficient is 0 (zero). In this case, it is more efficientto directly use correlation values (S₁ to S₆) because it is actuallyunnecessary to find each of the coefficients (a to f). Expressions (3)to be solved are as follows. Note here that an origin serves as a targetwindow standard.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{{x = \frac{{2 \times S_{3} \times S_{4}} - {S_{5} \times S_{2}}}{S_{2}^{2} - {4 \times S_{1} \times S_{3}}}}{y = \frac{{2 \times S_{1} \times S_{5}} - {S_{2} \times S_{4}}}{S_{2}^{2} - {4 \times S_{1} \times S_{3}}}}} & {{Expression}\mspace{14mu}(3)}\end{matrix}$

Note that such determination of positional offset by use of thesub-pixel-level accuracy is carried out in at least one region,desirably in several regions.

Then, the captured image determination section 102 supplies a determinedresult as to whether or not the process execution requirements are met.

(4) Notification to User

In response to the determined result received from the captured imagedetermination section 102, the control section 109 controls the displaysection 105 to display a message urging image capture to be carried outagain.

For example, when receiving, from the captured image determinationsection 102, a determined result that an angle of the skew θ fallsoutside the given range, the control section 109 controls the displaysection 105 to display a message which urges image capture to be carriedout again so that the image capture object is not skewed.

In response to a determined result that features indicative of a degreeof the geometric distortion (here, a ratio between the lengths of theopposite sides of the image capture object in the captured image) fallsoutside the given range, the control section 109 controls the displaysection 105 to display a message which urges image capture to be carriedout again from the normal direction of the plane of the image captureobject on which plane the text image is formed.

Further, in response to a determined result that the number of capturedimages which are offset by a given amount falls below a given number,the control section 109 controls the display section 105 to display amessage, urging image capture to be carried out again, such as “Thisimage may not be well processed. Please carry out image capture again.”so that a new image is obtained. Then, the captured image determinationsection 102 carries out the determination processes with respect to aplurality of newly captured images again. When it is determined that allthe process execution requirements are met, the plurality of newlycaptured images are used in processes carried out at subsequentprocesses. Alternatively, it is possible to use, in the subsequentprocesses, a captured image which meets all the process executionrequirements, by causing the captured image determination section 102 tocarry out the determination processes with respect to a combination ofthe image which previously captured and an image captured again.

(5) Arrangement of the Image Output Apparatus

An arrangement of the image output apparatus 200 is described below withreference to FIGS. 10 through 13. In the present embodiment, the imageoutput apparatus 200 is a multifunction printer which has functions of ascanner, a printer, a copying machine, and the like.

FIG. 10 is a block diagram illustrating the arrangement of the imageoutput apparatus 200. The image output apparatus 200 includes an imagescanning section 201, an image processing section (a correctionprocessing section) 202, a certifying section 203, an image formingsection (output section) 204, a display section 205, an input section206, a first communication section (receiving section) 207, a secondcommunication section (an output section) 208, a recording mediumaccessing section 209, a storage section 210, and a control section (anoutput section) 212.

The image scanning section 201 scans a document and has a scannersection including a CCD (Charge Coupled Device) which converts lightreflected from the document to an electric signal (an analogue imagesignal) which has been subjected to R, G, and B color separations. Then,the image scanning section 201 supplies this electric signal.

The image processing section 202 carries out given image processing withrespect to image data. According to the present embodiment, the imageprocessing section 202 carries out the image processing including thehigh resolution correction with respect to the captured image datatransmitted from the portable terminal apparatus 100. The imageprocessing carried out by the image processing section 202 with respectto the captured image data will be described later in detail.

The certifying section 203 carries out user certification when theoutput process is carried out with respect to the captured image datareceived from the portable terminal apparatus 100. In detail, thecertifying section 203 carries out the user certification by comparing(a) the user information received from the portable terminal apparatus100 with (b) the user information entered from the input section 206 (auser ID and a password). The certifying section 203 transmits acertified result to the control section 212.

The image forming section 204 forms an image on recording paper such aspaper by use of an electrophotographic printing method, an ink-jetmethod, or the like. Namely, the image forming section 204 carries outthe printing process which is one of the output processes.

The display section 205 is realized by a liquid crystal display device,for example. The input section 206 is provided for entering data by, forexample, touching a touch panel or a button included in the liquidcrystal display device.

The first communication section 207 has functions of the serial/paralleltransfer and the wireless data communication which are carried out inconformity with the USB 1.1 or USB 2.0 Standard. The first communicationsection 207 receives, from the portable terminal apparatus 100, thecaptured image data to which the file name, the information on the modelof the portable terminal apparatus 100, the user information, and theoutput process information are added.

The second communication section 208 has the following functions (a)through (c): (a) data communication employing a wireless technologywhich is in conformity with any one of LAN standards IEEE 802.11a, IEEE802.11b, and IEEE 802.11g, (b) data communication with a network, via aLAN cable, having a communications interface function employing Ethernet(registered trademark), and (c) data communication employing a wirelesstechnology which is in conformity with any one of communication systemssuch as IEEE 802.15.1 (so-called Bluetooth (registered trademark) whichis the wireless communication standard, the infrared communicationstandard such as IrSimple, and Felica (registered trademark).

The second communication section 208 carries out, as the output process,(i) the filing process for causing the captured image data which hasbeen subjected to the high resolution correction to be stored in theserver or (ii) the e-mail transmission process for transmitting thee-mail to which the captured image data which has been subjected to thehigh resolution correction is attached.

The recording medium accessing section 209 reads out a program from arecording medium in which the program is recorded. The storage section210 serves as a section in which a program for causing the sections ofthe image output apparatus 200 to carry out their respective processesis stored.

The control section 212 carries out control with respect to the sectionsincluded in the image output apparatus 200. In detail, when the firstcommunication section 207 receives the plurality of pieces of capturedimage data from the portable terminal apparatus 100, the control section212 supplies the plurality of pieces of captured image data to the imageprocessing section 202 so as to control the image processing section 202to carry out the image processing. In addition, the control section 212supplies, to the certifying section 203, the user information added tothe image data so as to control the certifying section 203 to carry outa certification process. When receiving a certified result that thecertification has been successfully carried out, the control section 212controls the corresponding process to be carried out in accordance withthe output process information added to the captured image data. Namely,in a case where the output process information is indicative of theprinting process, the control section 212 controls the image formingsection 204 to carry out the printing in accordance with the capturedimage data which has been subjected to the image processing by the imageprocessing section 202. Alternatively, in a case where the outputprocess information is indicative of the filing process or the e-mailtransmission process, the control section 212 controls the secondcommunication section 208 to carry out the filing process or the e-mailtransmission process in accordance with the captured image data whichhas been subjected to the image processing by the image processingsection 202.

(6) Image Processing carried out by the Image Processing Section

The image processing carried out by the image processing section 202 isdescribed below in detail. Note that the description below discussesdetails of the image processing carried out with respect to theplurality of pieces of captured image data received from the portableterminal apparatus 100, though the image processing section 202 alsocarries out the image processing with respect to the image data scannedby the image scanning section 201.

According to the present embodiment, the image processing section 202carries out the geometric distortion correction, the lens distortioncorrection, and the high resolution correction, with respect to theplurality of pieces of captured image data transmitted from the portableterminal apparatus 100. Processes carried out in the respectivecorrections are described below.

(6-1) Geometric Distortion Correction and Lens Distortion Correction

Like the captured image determination section 102, the image processingsection 202 sequentially detects, by the raster scanning, points on anedge of the image capture object in the captured image. Then, the imageprocessing section 202 carries out a curve fitting with respect to thepoints detected on the edge, and carries out the lens distortioncorrection based on a curvilineal expression.

In detail, the image processing section 202 detects the edge points ofthe detected image capture object and classifies, like the capturedimage determination section 102, the edge points into four groups whichcorrespond to four sides of the image capture object (see solid lines inFIG. 12). Subsequently, the image processing section 202 carries out aquadratic curve approximation with respect to the edge points whichbelong to each of the four groups. Four quadratic curves thus determinedwith respect to the respective four groups correspond to the respectivefour sides of the image capture object. In addition, the imageprocessing section 202 finds four intersections of the four quadraticcurves which intersections correspond to corner sections of a regiondefined by the four quadratic curves. Next, the image processing section202 finds a bound box (see one-dot chain lines in FIG. 12) in which thefour quadratic curves found for the respective four sides arecircumscribed, and which is similar to a quadrangle (see dotted lines inFIG. 12) defined by connecting the four intersections. Then, the imageprocessing section 202 carries out a transformation with respect to thelocation of pixels in a region where the image capture object is locatedin the captured image so that the edge pixels of the image captureobject which has been corrected are located on the sides of the boundbox. Such a transformation can be carried out by carrying outcalculations in accordance with vectors from a reference point (e.g.,the centroid of the region where the image capture object is located).This allows the lens distortion, due to the image capture section 101 ofthe portable terminal apparatus 100, to be corrected.

Further, the image processing section 202 carries out the geometricdistortion correction as below. The image processing section 202 cancarry out a similar mapping transformation with respect to the boundbox, which has been found as described above, in accordance with anaspect ratio (e.g., 7:10 in the case of A-/B-size used when outputting abusiness document) of the image capture object. A publicly-knowntechnique can be used as the mapping transformation (see FIG. 13). Notethat the image processing section 202 can carry out the mappingtransformation in accordance with an aspect ratio stored in the storagesection 210 or an aspect ratio entered from the input section 206.

Note that methods for correcting the geometric distortion and the lensdistortion are not limited to the above methods and that publicly-knowntechniques can be employed for the correction.

(6-2) High Resolution Correction by use of a Plurality of Pieces ofImage Data

As described earlier, the image processing section 202 receives, fromthe portable terminal apparatus 100, the plurality of pieces of the dataof the images captured with respect to the identical image captureobject. Then, the image processing section 202 carries out the highresolution correction in accordance with the plurality of pieces of thecaptured image data.

As for a method for forming a high resolution image in accordance with aplurality of pieces of image data, several methods are disclosed in theJournal of the Institute of Image Information and Television EngineersVol. 62, No. 3, pp. 337 through 342 (published in 2008). Generally, thehigh resolution correction process includes a positioning process for aplurality of images and a reconstructing process. In the presentembodiment, the normalized correlation pattern matching (see thedescription of (3-3)) is used as an example of a positioning process.Namely, it is possible to carry out the positioning for a plurality ofimages by displacing the plurality of images by an offset amountcorresponding to an extreme value of the foregoing S(x, y).

Next, the image processing section 202 carries out the reconstructingprocess. Namely, the image processing section 202 prepares reconstructedimage data whose number of pixels corresponds to a magnificationobtained after the resolution conversion. Note, however, that areconstructed image is assumed to have a size identical to that of thecaptured image. Then, the image processing section 202 determines pixelvalues of respective pixels in the reconstructed image data. Namely, theimage processing section 202 selects, from the plurality of capturedimages, a plurality of pixels of the captured image (captured imagepixels) located in the vicinity of each of the pixels (reconstructedpixels) in the reconstructed image data, and then carries out aninterpolation with respect to the reconstructed pixel in accordance witha general interpolation method (e.g., a linear interpolation method anda bi-cubic interpolation method).

In detail, two captured image pixels located in the vicinity of a targetreconstructed pixel are selected in each of transverse and longitudinaldirections. For example, two captured image pixels, whose line segment(see the dotted lines in FIG. 14) is the closest to the targetreconstructed pixel, are selected. Assume here that the two capturedimage pixels selected in the transverse direction are a captured imagepixel 1-2 (pixel value: V_(i1-2): pixel values of the following capturedimage pixels will be similarly indicated) of a first captured image anda captured image pixel 1-4 of the first captured image, whereas the twocaptured image pixels selected in the longitudinal direction are acaptured image pixel 2-1 of a second captured image and a captured imagepixel 2-2 of the second captured image. Note that it is assumed that thecaptured image pixels located in the vicinity of the reconstructed pixelare selected from the plurality of pieces of captured image data whichhave been subjected to the geometric distortion correction and the lensdistortion correction. This makes it possible to carry out the highresolution correction in a state where the geometric distortion and thelens distortion have already been corrected. Alternatively, a coordinatevalue obtained after the correction can be found by taking intoconsideration the geometric distortion correction and the lensdistortion correction for the uncorrected plurality of pieces ofcaptured image data. Namely, it is possible to (i) carry out thereconstruction process after only finding correction values of thegeometric distortion and the lens distortion, and then (ii) carry outthe coordinate transformation by use of the correction values.

Subsequently, two intersections of (i) the line segments each of whichis defined by the two points selected in the transverse and longitudinaldirections and (ii) straight lines on each of which the targetreconstructed pixel is located and each of which is perpendicular to acorresponding one of the line segments are found. In a case where thetwo intersections are internally dividing points of t:1−t and u:1−u onthe respective two line segments (see FIG. 14), the image processingsection 202 can find a pixel value V_(s) of the target reconstructedpixel in accordance with the following expression (4). It follows thatthe linear interpolation is carried out. Then, pixel values of all thereconstructed pixels are similarly found, so that it is possible toprepare reconstructed image data which has been subjected to the highresolution correction.

[Math. 5]V _(S)={(1−t)V _(i1−2) +tV _(i1−4)+(1−u)V _(i2−1) +uV _(i2−2)}/2  Expression (4)

Note that an alternative interpolation method can be employed. Note alsothat a further method disclosed in the Journal of the Institute of ImageInformation and Television Engineers Vol. 62, No. 3, pp. 337 through 342(published in 2008) can be employed. For example, it is possible toemploy an interpolation method such as a MAP (Maximum A Posteriori)method in which an assessment function which corresponds to anassumptive posterior probability is first minimized so that the pixelvalues of all the reconstructed pixels are found.

(7) Image Processing Method carried out in the Captured Image ProcessingSystem

A flow of processes carried out in the captured image processing systemis described below with reference to FIG. 1. Note that (a) of FIG. 1illustrates a processing flow in the portable terminal apparatus 100,and (b) of FIG. 1 illustrates a processing flow in the image outputapparatus 200.

First, the portable terminal apparatus 100 receives a selection of animage capture mode. Here, the user selects the text image capture modein a case where the user (i) carries out image capture with respect to arectangular image capture object shape such as paper on which a textimage is printed or a display screen on which the text image isdisplayed and (ii) wishes the image output apparatus 200 to output acaptured image which has been subjected to the high resolutioncorrection.

In the portable terminal apparatus 100 which has received the selectionof the text image capture mode, the control section 109 controls thedisplay section 105 to display a window which urges the user to enter,from the input section 106, a magnification of resolution conversionrequired for the high resolution correction. This allows the controlsection 109 to obtain the magnification which the user entered from theinput section 106. The control section 109 determines, in accordancewith the magnification, (i) the number of times of image capture carriedout by the image capture section 101 and (ii) a part of the processexecution requirements which is used in the captured image determinationsection 102. Further, the control section 109 controls the displaysection 105 to display a window which urges the user to enter, from theinput section 106, (i) a kind of an output process and (ii) a settingrequirement for carrying out the output process. This allows the controlsection 109 to obtain output process information entered from the inputsection 106.

When detecting a shutter click, the image capture section 101consecutively carries out image capture as many times as the controlsection 109 sets (S10).

Next, the image processing section 103 carries out at least the A/Dconversion process with respect to a plurality of pieces of capturedimage data (multiple captured image data). Then, the captured imagedetermination section 102 determines whether or not the plurality ofpieces of captured image data which have been subjected to the A/Dconversion process meet the process execution requirements (S11). Notethat details of how to determine and items to be determined are asdescribed earlier, for example, in (3-1) through (3-3).

In a case where the captured image determination section 102 determinesthat no process execution requirements are met (NO in S11), the controlsection 109 controls the display section 105 to display a message urgingimage capture to be carried out again, so that the user is notified ofthe message (S12). In a case where even an image which has been capturedagain meets no determination items as mentioned above, the portableterminal apparatus 100 repeatedly carries out steps S10 through S12.

In contrast, in a case where the captured image determination section102 determines that the process execution requirements are met (YES inS11), the control section 109 assigns file names to the respectiveplurality of pieces of captured image data which meet the processexecution requirements (S13). Note that the control section 109 canautomatically assign, to the respective plurality of pieces of capturedimage data, (a) different file names (e.g., serial numbers which vary inaccordance with image capture date and time) or (b) file names enteredfrom the input section 106. Thereafter, the control section 109transfers to the communication section 104 the image data, to which thefile names are assigned, together with (i) the information on the modelof the portable terminal apparatus 100 and the user information whichare stored in the storage section 108 and (ii) the output processinformation. Then, the communication section 104 transmits these piecesof information to the image output apparatus 200 (S14).

Note that the control section 109 temporarily can save, in the storagesection 108 or a memory card, the image data to which the file names areassigned, and then transfer the image data to the communication section104 and to the image output apparatus 200, together with the informationon the model of the portable terminal apparatus 100, the userinformation, and the output process information.

Next, the first communication section 207 of the image output apparatus200 receives, from the portable terminal apparatus 100, the plurality ofpieces of captured image data, the information on the model of theportable terminal apparatus 100, the user information, and the outputprocess information (S20). After receiving the pieces of information anddata, the image processing section 202 carries out the geometricdistortion correction and the lens distortion correction (S21) (see thedescription of (6-1), for example). Further, the image processingsection 202 carries out the high resolution correction in accordancewith the plurality of pieces of captured image data (S22) (see thedescription of (6-2), for example). Thereafter, the control section 212controls the storage section 210 to store the captured image dataprocessed by the image processing section 202 (S23).

Subsequently, the control section 212 determines whether or not anoutput instruction to output the captured image data is entered from theinput section 206 (S24). In a case where no output instruction isentered (NO in S24), the control section 212 waits for the outputinstruction to be entered.

In contrast, in a case where the output instruction is entered (YES inS24), the certifying section 203 controls the display section 205 todisplay a window urging user information (such as a user ID or apassword) to be entered from the input section 206. This allows the userinformation to be entered from the input section 206. Then, thecertifying section 203 carries out user certification (S25). Note thatthe certifying section 203 can alternatively obtain the user informationfrom a non-contact IC card possessed by the user with the use of anon-contact IC card reader/writer included in the image output apparatus200.

During the user certification, the certifying section 203 (i) comparesthe user information entered from the input section 206 with the userinformation received from the portable terminal apparatus 100 and (ii)determines whether or not these pieces of information match each other(S26). In a case where the image output apparatus 200 receives, from theportable terminal apparatus 100, the user information which matches theuser information entered from the input section 206 (YES in S26), thecontrol section 212 controls the output process to be carried out inaccordance with the output process information received from theportable terminal apparatus 100 (S27). For example, in a case where theoutput process information is indicative of the printing process, thecontrol section 212 supplies, to the image forming section 204, aninstruction to carry out the printing process. Thereafter, theprocessing flow is ended.

In contrast, the user information entered from the input section 206does not match the user information received from the portable terminalapparatus 100 (NO in S26), the certifying section 203 determines whetheror not the certification is carried out not less than a given times(S28). In a case where the certification is carried out less than thegiven times (NO in S28), S25 and S26 processes are repeated. In a casewhere the certification is carried out not less than the given times(YES in S28), the processing flow is ended with no output.

As described above, according to the present embodiment, it is possibleto cause the image output apparatus 200 to output the captured imagecaptured by the portable terminal apparatus 100 in a state where thecaptured image has been subjected to the high resolution correction.This allows an improvement in text readability. Therefore, even in acase where it is difficult to bring writing paper to the image outputapparatus 200, it is possible to obtain an image on the writing paperwith high resolution by carrying out image capture with respect to thewriting paper. Even in a case of a screen on which an image is displayedby a projector or the like, it is possible to later obtain a highresolution image from the image output apparatus 200, by causing theportable terminal apparatus 100 to carry out image capture with respectto the screen.

(8) Modifications

The captured image processing system of the present invention is notlimited to the description of the embodiment above, but can be variouslymodified. An example of a modified embodiment is described below.

(8-1) Number of Times of Image Capture

According to the above description, the control section 109 of theportable terminal apparatus 100 causes the image capture section 101 tocarry out image capture as many times as the number of the capturedimages required for the high resolution correction. Namely, the controlsection 109 sets, (i) the number of times of image capture and (ii) thenumber of pieces of captured image data which is required in the processexecution requirements, so that the number (i) is equal to the number(ii). However, the control section 109 can set the number (i) and thenumber (ii) so that the number (i) is greater than the number (ii). Forexample, in the case where the magnification of the resolutionconversion is set to ×2, it is possible to set the number of times ofimage capture to “3” while the number of pieces of captured image datarequired in the process execution requirements is set to “2”.

In a case where the number of times of image capture is thus larger thanthe number of pieces of captured image data required in the processexecution requirements, it is necessary for the captured imagedetermination section 102 to only determine whether or not there exists,in the captured image data whose number is equal to the number of timesof image capture, a combination of captured image data which meets theprocess execution requirements. For example, in a case where the numberof times of image capture is “3” while the required number is “2”, thereexist three combinations of the required number of pieces of thecaptured image data in the three pieces of captured image data. In thiscase, the captured image determination section 102 sequentiallydetermines whether or not the three combinations meet the processexecution requirements. When confirming a combination which meets theprocess execution requirements, the captured image determination section102 finishes the determination processes. Then, the communicationsection 104 transmits, to the image output apparatus 200, the capturedimage data included in the combination. Alternatively, the capturedimage determination section 102 can determine whether or not all thecombinations meet the process execution requirements. In this case, whena plurality of the combinations meet the process execution requirements,the control section 109 can select the combination whose detected offsetamount is the closest to a median of a given range, and transmits thecombination to the image output apparatus 200. For example, in a casewhere the given range is 0.3 to 0.7, the combination whose offset amountis the closest to 0.5 can be selected.

(8-2) Items Determined by the Captured Image Determination Section

According to the above description, the captured image determinationsection 102 determines (i) the skew and (ii) the geometric distortion ofthe image capture object in the captured image and (iii) an amount ofoffset of adjacent ones of the plurality of captured images, when thetext image capture mode is selected. However, items which the capturedimage determination section 102 determines are not limited to the items(i) through (iii). Examples of modified items to be determined aredescribed below.

(a) For example, the user may wish the image output apparatus 200 tooutput a captured image of not only the image capture object such aspaper on which a text image is printed or a display screen on which thetext image is displayed but also scenery or a figure which capturedimage has been subjected to the high resolution correction. In view ofthis, the portable terminal apparatus 100 can have, instead of orseparately from the text image capture mode, a high resolution outputmode in which the image output apparatus 200 outputs the captured imagethat has been subjected to the high resolution correction. In a case thehigh resolution output mode is provided separately from the text imagecapture mode, the portable terminal apparatus 100 can supply thecaptured image data to which information indicative of a selected modeis added. This allows the image output apparatus 200 to carry out itsoutput process in accordance with a selected mode.

In a case where the high resolution output mode is selected, thecaptured image determination section 102 of the portable terminalapparatus 100 only carries out the determination of an offset amount(see the description of (3-3)) but does not carry out the determinationof the skew and the geometric distortion of the image capture object(see the descriptions of (3-1) and (3-2)). Similarly, in the imageoutput apparatus 200 which is notified that the high resolution outputmode is selected, the image processing section 202 carries out the highresolution correction (see the description of (6-2)) but does not carryout the geometric distortion correction and the lens distortioncorrection (see the description of (6-1)). This also brings about aneffect of obtaining a higher resolution image than an actually capturedimage.

Note, in this case, that the captured image determination section 102needs to extract, from the vicinity of the center of the captured image,an offset detecting partial image.

(b) Other items can be determined in addition to the above item.Examples of the other items include a brightness, a contrast, a colorbalance, and a blur (an intense camera shake).

As for a brightness, for example, in a case where overexposure occurs(the captured image is too bright) or underexposure occurs (the capturedimage is too dark), image capture may be required to be carried outagain. In view of this, the captured image determination section 102finds, for example, maximum and minimum ones of pixel values obtained inthe captured image data. In a case where the maximum value is not morethan a given threshold (e.g., 100 in case of 8 bits), the captured imagedetermination section 102 determines that underexposure occurs, and thensupplies, to the control section 109, a determined result. In contrast,in a case where the minimum value is not less than a given threshold(e.g., 150 in case of 8 bits), the captured image determination section102 determines that overexposure occurs, and then supplies, to thecontrol section 109, a determined result. Then, in response to thedetermined result that underexposure or overexposure occurs, the controlsection 109 controls the display section 105 to display the determinedresult and an instruction urging image capture to be carried out again.Alternatively, the control section 109 changes the setting of the imagecapture section 101 so that the image capture section 101 has longerexposure time in the case of underexposure. In contrast, the controlsection 109 changes the setting of the image capture section 101 so thatthe image capture section 101 has shorter exposure time in the case ofoverexposure. Thereafter, the control section 109 can notify the user ofthe instruction urging image capture to be carried out again.

As for a contrast, in a case where a difference between the maximum andminimum values of the pixel values obtained in the captured image datais not more than a given threshold, the captured image determinationsection 102 determines that the captured image has a poor contrast.Then, in response to a determined result that the captured image has apoor contrast, the control section 109 controls the display section 105to display the determined result and an instruction urging image captureto be carried out again.

Note that the captured image determination section 102 can carry out thedetermination of a brightness and a contrast with respect to each ofcolor channels or can use an average value (R+G+B/3) or a brightnessvalue (0.299×R+0.587×G+0.114×B: conforming to NTSC).

As for a color balance, it is possible to detect an occurrence of anexcessive imbalance in a given color channel by comparing average valuesof or maximum/minimum values of the respective color channels (R, G, andB). In view of this, the captured image determination section 102determines that the captured image has a poor color balance, forexample, in a case where (i) average values (Ra, Ga, and Ba) of thepixel values of the respective color channels which pixel values areobtained in the captured image data and have values in the vicinity of amaximum brightness value (in a range of maximum brightness to (maximumbrightness −5)) are found, and (ii) a difference between the maximumvalue and the minimum value of average values (Ra, Ga, and Ba) of therespective color channels is not less than a corresponding given value[Max(Ra, Ga, and Ba)−Min(Ra, Ga, and Ba)>0.1×Max(Ra, Ga, and Ba)]. Then,in response to the determined result that the captured image has a poorcolor balance, the control section 109 causes the display section 105 todisplay the determined result and an instruction urging image capture tobe carried out again.

As for a blur (an intense camera shake: a so-called motion blur), anedge of the captured image is less acute when the blur occurs. In viewof this, the captured image determination section 102 prepares an edgeintensity image by use of the edge extraction filter (see FIG. 9), andprepares a histogram so as to find a standard deviation of the histogram(a square root of the variance). In a case where the standard deviationis not more than a given threshold (e.g., 5), the captured imagedetermination section 102 determines that a blur occurs in the capturedimage. Then, in response to a determined result of the determinationthat a blur occurs in the captured image, the control section 109 causesthe display section 105 to display the determined result and aninstruction urging image capture to be carried out again.

(8-3) Processing Items in the Image Output Apparatus

The above description discusses the case where the image processingsection 202 of the image output apparatus 200 carries out the geometricdistortion correction, the lens distortion correction, and the highresolution correction with respect to the plurality of pieces ofcaptured image data received from the portable terminal apparatus 100.However, the processing carried out by the image processing section 202is not limited to the above three corrections. The image processingsection 202 can further carry out corrections such as a color balancecorrection, a contrast correction, and a skew correction, in addition tothe above three corrections.

A color balance can be corrected in accordance with a method in whichthe image processing section 202 finds maximum and minimum values of thereceived captured image data for each of the color channels, prepareslook-up tables which cause the color channels to have uniform maximumand minimum values, and apply the look-up tables to the respective colorchannels. FIG. 11 shows an example of the look-up tables. As shown inFIG. 11, in a case where (i) a given channel has a maximum value of MXand a minimum value of MN and (ii) the data has 8 bits, a look-up tablecan be prepared that causes an increase from MN in increments of(MX−MN)/255.

The image processing section 202 carries out the contrast correction ina similar manner to the color balance correction. Note that the look-uptables applied to the respective color channels can be identical in acase where it is unnecessary to change a color balance to a specificone.

Note that an alternative publicly-known technique can be applied to thecolor balance and contrast corrections.

Further, the image processing section 202 can detect the skew of theimage capture object by the method disclosed in (3-1) and carry out aprocess for rotating the captured image data so that the skew has anangle of 0 (zero) degree. Alternatively, the portable terminal apparatus100 can transmit, to the image output apparatus 200, an angle of theskew which angle is detected in (3-1), together with the captured imagedata. Then, the image processing section 202 can carry out the processfor rotating the captured image data in accordance with the angle of theskew which angle is received from the portable terminal apparatus 100 sothat the skew of the captured image has the angle of 0 (zero) degree.

(8-4) Image Capture Section

The above description discusses an arrangement employing the fact that acamera shake, occurring when the image capture section 101 consecutivelycarries out image capture a plurality of times, causes offset of theplurality of captured images. However, the embodiment is not limited tothis. Instead, the image capture section 101 can slightly displace animage capture device (CCD/CMOS) or a lens when consecutively carryingout the image capture the plurality of times. This securely causesoffset of the plurality of captured images.

(8-5) Output Process Information

The above description discusses an arrangement in which the portableterminal apparatus 100 obtains and transmits the output processinformation to the image output apparatus 200. However, the embodimentis not limited to this. The image output apparatus 200 can obtain theoutput process information (the information indicative of the kind ofthe output process and the setting requirement for the output process)when obtaining the user information so as to carry out the usercertification.

(8-6) Output Process

Before carrying out the filing process or the e-mail transmissionprocess, the control section 212 of the image output apparatus 200 canconvert, to a high-compression PDF, the captured image data processed bythe image processing section 202. Note that the high-compression PDFrefers to PDF data in which the image data is separated into abackground part and a text part and optimum compression processes arecarried out with respect to the respective parts. This allows favorablereadability and a reduction in size of an image file.

Alternatively, before carrying out the filing process or the e-mailtransmission process, the control section 212 can carry out an OCR(Optical Character Recognition) process with respect to the capturedimage data processed by the image processing section 202 so as toprepare text data. The control section 212 can convert the capturedimage data to a PDF, and then add the text data to the PDF as atransparent text. Note that the transparent text is data forsuperimposing (embedding) a recognized text on (in) the image data astext information so that the recognized text is apparently invisible.For example, an image file in which a transparent text is added to imagedata is generally used in a PDF file. Then, the control section 212 cancause PDF data, to which the prepared transparent text is added, to beoutputted. This allows an output of an electronic document easy toutilize as if it were a file in which a text search can be carried out.

(8-7) Image Processing Section of the Image Output Apparatus

The above description discusses an arrangement in which the imageprocessing section 202 of the image output apparatus 200 carries outcorrections including the high resolution correction. Instead, the imageoutput apparatus 200 can cause a server including an image processingsection 202 to carry out, with respect to the captured image data, thehigh resolution correction and the other image processing such as thegeometric distortion correction, the lens distortion correction, thecontrast correction, and the color balance correction. Note, in thiscase, that the server will serve as an image output apparatus forcarrying out the high resolution correction with respect to the capturedimage data received from the portable terminal apparatus 100, and foroutputting the captured image data which has been subjected to the highresolution correction.

(9) Program and Recording Medium

The present invention can be achieved by recording, on acomputer-readable recording medium in which a program to be executed bya computer is recorded, a method in which the image captured by theportable terminal apparatus 100 is transmitted to and outputted by theimage output apparatus 200.

This makes it possible to portably provide a recording medium in whichprogram codes (an executable program, an intermediate code program, anda source program) for carrying out the above process are recorded.

Note, in the present embodiment, that the recording medium can be amemory (not illustrated) such as a ROM or the recording medium itselfcan be a program medium (not illustrated) because the process is carriedout by a microcomputer. Alternatively, the recording medium can be aprogram medium from which the program codes can be read out by carryingout loading of a recording medium with respect to a program readingdevice provided as an external storage apparatus (not illustrated).

In any case, an arrangement can be employed in which a stored program isexecuted by access of a microprocessor. Alternatively, in any case, asystem can be employed in which the program codes are read out anddownloaded on a program storage area (not illustrated) of themicrocomputer, and then the program is executed. The program for thedownloading is stored in a main body in advance.

Note here that the program medium is a recording medium which isarranged to be detachable from the main body. The program media can alsobe a medium fixedly bearing a program code which medium includes (i) atape such as a magnetic tape or a cassette tape, (ii) a disk including amagnetic disk such as a flexible disk or a hard disk and an optical disksuch as a CD-ROM, an MO, an MD, or a DVD, (iii) a card, such as an ICcard (including a memory card) or an optical card, or (iv) asemiconductor memory of a mask ROM, EPROM (Erasable Programmable ReadOnly Memory), EEPROM (Electrically Erasable Programmable Read OnlyMemory), or a flash ROM.

Further, the present embodiment has a system architecture which isconnectable to a communication network including the Internet. As such,the recording medium can be a medium which to bears the program codes ina flexible manner so that the program code is downloaded from thecommunication network. Note that, in a case where the program isdownloaded from the communication network as described above, theprogram for the downloading can be stored beforehand in the main body orcan be installed from an alternative recording medium. Note that thepresent invention can also be realized in a form of a computer datasignal in which the program code is embodied by an electronictransmission and which is embedded in carrier waves.

The recording medium is read by a program scanning device included inthe portable terminal apparatus 100 or the image output apparatus 200,whereby the image processing method is carried out.

As described earlier, a portable terminal apparatus of the presentinvention for transmitting image data to an image output apparatus, theimage output apparatus including: a correction processing section forcarrying out a high resolution correction in accordance with a pluralityof pieces of the image data so as to obtain high resolution image datawhich has a higher resolution than the plurality of pieces of the imagedata, the image output apparatus outputting the high resolution imagedata corrected by the correction processing section, said portableterminal apparatus, includes: an image capture section for consecutivelymore than once capturing images of an identical object; a captured imagedetermination section for determining whether or not a plurality ofpieces of captured image data captured by the image capture section meeta given requirement; and a transmission section for transmitting, to theimage output apparatus, captured image data which is determined, by thecaptured image determination section, to meet the given requirement, thegiven requirement including a requirement A, the requirement A beingsuch that the plurality of pieces of captured image data consecutivelycaptured by the image capture section include a combination of a givennumber (an integer not less than 2) of pieces of captured image datawhich are applicable to be subjected to the high resolution correctionby the correction processing section and are offset by a given amount,the transmission section transmitting the given number of pieces ofcaptured image data which are determined to be the combination by thecaptured image determination section.

A method of the present invention for controlling a portable terminalapparatus for transmitting image data to an image output apparatus, theimage output apparatus including: a correction processing section forcarrying out a high resolution correction in accordance with a pluralityof pieces of the image data so as to obtain high resolution image datawhich has a higher resolution than the plurality of pieces of the imagedata, the image output apparatus outputting the high resolution imagedata corrected by the correction processing section, the portableterminal apparatus including an image capture section for consecutivelymore than once capturing images of an identical object, said methodincludes the steps of: (a) determining whether or not a plurality ofpieces of captured image data captured by the image capture section meeta given requirement; and (b) transmitting, to the image outputapparatus, captured image data which is determined, by the capturedimage determination section, to meet the given requirement, the givenrequirement including a requirement A, the requirement A being such thatthe plurality of pieces of captured image data consecutively captured bythe image capture section include a combination of a given number (aninteger not less than 2) of pieces of captured image data which areapplicable to be subjected to the high resolution correction by thecorrection processing section and are offset by a given amount, in thestep (b), transmitting the given number of pieces of captured image datawhich are determined to be the combination in the step (a).

According to the arrangement, the portable terminal apparatus determineswhether or not the plurality of pieces of captured image dataconsecutively captured by the image capture section include acombination of a given number (an integer not less than 2) of pieces ofcaptured image data which are applicable to be subjected to the highresolution correction by the correction processing section and areoffset by a given amount. In a case where the plurality of pieces ofcaptured image data include the combination, the portable terminalapparatus transmits the given number of pieces of captured image datawhich are determined to be the combination.

This causes the portable terminal apparatus to transmit, to the imageoutput apparatus, only the given number of pieces of captured image datawhich can be subjected to the high resolution correction by the imageoutput apparatus. This allows a user to easily obtain, from an imageoutput apparatus, a high resolution image which is an image captured bya portable terminal apparatus and has a higher resolution than thecaptured image. According to this, for example, in a case where imagecapture is carried out with respect to an image capture object includinga text image, a text included in the text image is enhanced inreadability in an image obtained from the image output apparatus.

The portable terminal apparatus of the present invention is preferablyarranged to further include a notification section for notifying a userof need to capture images again when the captured image determinationsection has determined that no given requirement is met.

According to the arrangement, the user can easily grasp that it isnecessary to carry out image capture again so as to obtain a highresolution image from the image output apparatus. This makes it possibleto take an immediate step to carry out image capture.

The portable terminal apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has arectangular plane on which a text image is formed; and the givenrequirement further includes a requirement B, the requirement B beingsuch that features fall within a given range, the features indicative ofa degree of a distortion of the rectangular plane in the captured imagedata, the distortion occurring when the image capture section capturedimage data in a direction different from a normal direction of therectangular plane.

Note here that examples of the identical image capture object which hasthe rectangular plane include document paper or a poster on which a textimage is formed, a screen projected by a projector or the like, and ascreen included in a display device.

A variety of methods have been known as a method for correcting adistortion of the rectangular plane which distortion occurs in thecaptured image data due to the image capture which is carried out by theimage capture section from the direction different from the normaldirection of the rectangular plane. However, in a case where thedistortion occurs to a large degree, it is difficult to securely obtainan undistorted image even if such a correction is carried out. However,according to the arrangement, the portable terminal apparatus transmitsthe captured image data only in a case where the features indicative ofa degree of the distortion fall within the given range. For this reason,the captured image data transmitted to the image output apparatus has adistortion which is relatively small. This is because the rectangularplane is distorted to the degree that falls within the given range. Thisallows a secure correction for the distortion in the image outputapparatus.

The portable terminal apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has arectangular plane on which a text image is formed; and the givenrequirement further includes a requirement C, the requirement C beingsuch that a skew angle of the rectangular plane in the captured imagedata falls within a given range.

Assume that a rectangular object is skewed in a captured image. Even ifa high resolution image is obtained from an image output apparatus, adocument is also skewed in the high resolution image. However, accordingto the arrangement, the portable terminal apparatus transmits thecaptured image data only in a case where the angle of the skew fallswithin the given range. Note here that the given range refers to a rangeincluding 0°, for example, within 0±15°. Such a setting allows aprevention of an image which is steeply skewed from being transmitted tothe image output apparatus.

An image output apparatus of the present invention includes: a receivingsection for receiving, from a portable terminal apparatus as mentionedabove, the given number of pieces of captured image data obtained fromthe image capture which is consecutively more than once carried out bythe image capture section; a correction processing section for carryingout a high resolution correction for preparing, in accordance with thegiven number of pieces of captured image data received by the receivingsection, high resolution image data which has a higher resolution thanthe given number of pieces of captured image data; and an output sectionfor carrying out an output process in which the high resolution imagedata prepared by the correction processing section or an image indicatedby the high resolution image data is outputted.

An image output method of the present invention includes the steps of:(a) receiving, from a portable terminal apparatus as mentioned above,the given number of pieces of captured image data obtained from theimage capture which is consecutively more than once carried out by theimage capture section; (b) carrying out a high resolution correction forpreparing, in accordance with the given number of pieces of capturedimage data received in the step (a), high resolution image data whichhas a higher resolution than the given number of pieces of capturedimage data; and (c) carrying out an output process in which the highresolution image data prepared in the step (b) is outputted.

According to the arrangement, the image output apparatus obtains, fromthe portable terminal apparatus, the given number of pieces of capturedimage data which are applicable to be subjected to the high resolutioncorrection. This allows the image output apparatus to easily prepare thehigh resolution image data in accordance with the given number of piecesof captured image data. This also allows the high resolution image datato have a higher quality. As a result, it is possible for a user toeasily obtain data of an image which has a higher resolution than animage captured by a portable terminal apparatus. According to this, forexample, in a case where image capture is carried out by a portableterminal apparatus with respect to an image capture object including atext image, it is possible to obtain, from an image output apparatus, animage in which a text included in the text image is enhanced inreadability.

The image output apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has arectangular plane on which a text image is formed; and the correctionprocessing section (i) corrects a distortion of the rectangular plane inthe captured image data, the distortion occurring when the image capturesection captured the image data in a direction different from a normaldirection of the rectangular plane and (ii) carries out the highresolution correction.

According to the arrangement, a user can easily obtain an undistortedimage from an image output apparatus even if image capture is obliquelycarried out with respect to an image capture object.

The image output apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has arectangular plane on which a text image is formed; and the correctionprocessing section (i) corrects a lens distortion in which therectangular plane has a curved edge in captured image data obtained bythe image capture section and (ii) carries out the high resolutioncorrection.

According to the arrangement, a user can easily obtain, from an imageoutput apparatus, an image which has no lens distortion.

The image output apparatus of the present invention is preferablyarranged such that the output section carries out, as the outputprocess, a printing process in which an image which corresponds to thehigh resolution image data is formed on recording paper.

According to the arrangement, a user can easily obtain, from an imageoutput apparatus, a printed matter on which an image which has a higherresolution than a captured image is printed.

The image output apparatus of the present invention is preferablyarranged such that the output section carries out, as the outputprocess, a storing process in which the high resolution image data isstored in an external storage apparatus. Alternatively, the image outputapparatus of the present invention is preferably arranged such that theoutput section carries out, as the output process, an e-mailtransmission process in which an e-mail, to which the high resolutionimage data is attached, is transmitted.

According to the arrangement, a user can easily obtain, from an imageoutput apparatus, data of an image which has a higher resolution than acaptured image.

The image output apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has atext image; and the output section outputs the high resolution imagedata in a state where the high resolution image data is separated into(i) a region which corresponds to the text image and (ii) the otherregion and the regions are compressed at respective differentcompression ratios.

According to the arrangement, it is possible to cause the highresolution image data to have less capacity. This allows a reduction inload imposed on communication during the output.

The image output apparatus of the present invention is preferablyarranged such that: the identical object whose images are captured has atext image; and the output section extracts a text from a region of thehigh resolution image data which region corresponds to the text image,converts the text to text data, and then outputs the high resolutionimage data to which the text data is added.

According to the arrangement, a user can easily carry out a text searchby use of text data.

A captured image processing system of the present invention includes: aportable terminal apparatus as mentioned above; and an image outputapparatus as mentioned above for receiving a plurality of pieces ofimage data from the portable terminal apparatus.

According to the arrangement, a user can easily obtain a high resolutionimage which is an image captured by a portable terminal apparatus andhas a higher resolution than the image.

Note that it is possible to cause a computer to realize a portableterminal apparatus and an image output apparatus which are mentionedabove. In this case, (i) a program for causing a computer to realize theportable terminal apparatus and the image output apparatus by causingthe computer to operate as each section mentioned above and (ii) acomputer-readable storage medium in which the program is recorded areboth encompassed in the scope of the present invention.

The present invention is not limited to the description of theembodiments above, but may be altered by a skilled person within thescope of the claims. An embodiment based on a proper combination oftechnical means disclosed in different embodiments is encompassed in thetechnical scope of the present invention.

Industrial Applicability

The present invention is applicable to a captured image processingsystem for carrying out data communication between a portable terminalapparatus and an image output apparatus.

REFERENCE SIGNS LIST

-   -   100 Portable terminal apparatus    -   200 Image output apparatus    -   101 Image capture section    -   102 Captured image determination section    -   103 Image processing section    -   104 Communication section (Transmission section)    -   105 Display section (Notification section)    -   106 Input section    -   109 Control section (Transmission section, Notification section)    -   202 Image processing section (Correction processing section)    -   204 Image forming section (Output section)    -   205 Display section    -   206 Input section    -   207 First communication section (Receiving section)    -   208 Second communication section (Output section)    -   212 Control section (Output section)

1. A portable terminal apparatus for transmitting image data to an imageoutput apparatus, the image output apparatus including: a correctionprocessing section for carrying out a high resolution correction inaccordance with a plurality of pieces of the image data so as to obtainhigh resolution image data which has a higher resolution than theplurality of pieces of the image data, the image output apparatusoutputting the high resolution image data corrected by the correctionprocessing section, said portable terminal apparatus, comprising: animage capture section for consecutively more than once capturing imagesof an identical object; a captured image determination section fordetermining whether or not a plurality of pieces of captured image datacaptured by the image capture section meet a given requirement; and atransmission section for transmitting, to the image output apparatus,captured image data which is determined, by the captured imagedetermination section, to meet the given requirement, the givenrequirement including a requirement A, the requirement A being such thatthe plurality of pieces of captured image data consecutively captured bythe image capture section include a combination of a given number ofpieces of captured image data which are applicable to be subjected tothe high resolution correction by the correction processing section,with the given number of pieces of captured image data being an integernot less than 2, and the plurality of pieces of captured image data areoffset by a given pixel amount, the transmission section transmittingthe given number of pieces of captured image data which are determinedto be the combination by the captured image determination section,wherein: the identical object whose images are captured has arectangular plane on which a text image is formed; and the givenrequirement further includes a requirement B, the requirement B beingsuch that features fall within a given range, the features indicative ofa degree of a geometric distortion of the rectangular plane in thecaptured image data, the geometric distortion occurring when the imagecapture section captured image data obliquely with respect to a normaldirection of the rectangular plane.
 2. The portable terminal apparatusas set forth in claim 1, further comprising: a notification section fornotifying a user of need to capture images again when the captured imagedetermination section has determined that no given requirement is met.3. The portable terminal apparatus as set forth in claim 1, wherein: theidentical object whose images are captured has a rectangular plane onwhich a text image is formed; and the given requirement further includesa requirement C, the requirement C being such that a skew angle of therectangular plane in the captured image data falls within a given range.4. A captured image processing system comprising: a portable terminalapparatus; and an image output apparatus for receiving a plurality ofpieces of image data from the portable terminal apparatus, the portableterminal apparatus including: an image capture section for consecutivelymore than once capturing images of an identical object; a captured imagedetermination section for determining whether or not a plurality ofpieces of captured image data meet a given requirement; and atransmission section for transmitting, to the image output apparatus,captured image data which is determined, by the captured imagedetermination section, to meet the given requirement, the givenrequirement including a requirement A, the requirement A being such thatthe plurality of pieces of captured image data consecutively captured bythe image capture section include a combination of a given number ofpieces of captured image data which are applicable to be subjected tothe high resolution correction by the correction processing section,with the given number of pieces of captured image data being an integernot less than 2, and are offset by a given pixel amount, thetransmission section transmitting the given number of pieces of capturedimage data which are determined to be the combination by the capturedimage determination section, the image output apparatus including: areceiving section for receiving the given number of pieces of capturedimage data from the portable terminal apparatus; a correction processingsection for carrying out a high resolution correction for preparing, inaccordance with the given number of pieces of captured image datareceived by the receiving section, high resolution image data which hasa higher resolution than the given number of pieces of captured imagedata; and an output section for carrying out an output process in whichthe high resolution image data prepared by the correction processingsection or an image indicated by the high resolution image data isoutputted, wherein: the identical object whose images are captured has arectangular plane on which a text image is formed; and the givenrequirement further includes a requirement B, the requirement B beingsuch that features fall within a given range, the features indicative ofa degree of a geometric distortion of the rectangular plane in thecaptured image data, the geometric distortion occurring when the imagecapture section captured image data obliquely with respect to a normaldirection of the rectangular plane.
 5. The captured image processingsystem as set forth in claim 4, wherein: the identical object whoseimages are captured has a rectangular plane on which a text image isformed; and the correction processing section (i) corrects a distortionof the rectangular plane in the captured image data and (ii) carries outthe high resolution correction.
 6. The captured image processing systemas set forth in claim 4, wherein: the identical object whose images arecaptured has a rectangular plane on which a text image is formed; andthe correction processing section (i) corrects a lens distortion inwhich the rectangular plane has a curved edge in captured image dataobtained by the image capture section and (ii) carries out the highresolution correction.
 7. The captured image processing system as setforth in claim 4, wherein the output section carries out, as the outputprocess, a printing process in which an image which corresponds to thehigh resolution image data is formed on recording paper.
 8. The capturedimage processing system as set forth in claim 4, wherein the outputsection carries out, as the output process, a storing process in whichthe high resolution image data is stored in an external storageapparatus.
 9. The captured image processing system as set forth in claim8, wherein: the identical object whose images are captured has a textimage; and the output section outputs the high resolution image data ina state where the high resolution image data is separated into (i) aregion which corresponds to the text image and (ii) the other region andthe regions are compressed at respective different compression ratios.10. The captured image processing system as set forth in claim 8,wherein: the identical object whose images are captured has a textimage; and the output section extracts a text from a region of the highresolution image data which region corresponds to the text image,converts the text to text data, and then outputs the high resolutionimage data to which the text data is added.
 11. The captured imageprocessing system as set forth in claim 4, wherein the output sectioncarries out, as the output process, an e-mail transmission process inwhich an e-mail, to which the high resolution image data is attached, istransmitted.
 12. The captured image processing system as set forth inclaim 11, wherein: the identical object whose images are captured has atext image; and the output section outputs the high resolution imagedata in a state where the high resolution image data is separated into(i) a region which corresponds to the text image and (ii) the otherregion and the regions are compressed at respective differentcompression ratios.
 13. The captured image processing system as setforth in claim 11, wherein: the identical object whose images arecaptured has a text image; and the output section extracts a text from aregion of the high resolution image data which region corresponds to thetext image, converts the text to text data, and then outputs the highresolution image data to which the text data is added.
 14. Anon-transitory computer-readable recording medium in which a program isrecorded for causing a portable terminal apparatus recited in claim 1 tooperate, the program causing a computer to function as each section ofthe portable terminal apparatus.